removeUI does not remove UI completely

I have two tabItems in my dashboard as following in the ui.r file:

library(shiny)
library(shinydashboard)
library(shinyWidgets)


ui <- dashboardPage(
  dashboardHeader(
    title = "Title"
  ),

  dashboardSidebar(collapsed = FALSE,
                   sidebarMenu(id = 'sidebarMenu',
                               menuItem('Tab 1', tabName = 'tab1'),
                               menuItem('Tab 2', tabName = 'tab2')
                   )
  ),

  dashboardBody(
    tabItems(

      # First tab content
      tabItem(tabName = "tab1",

              div(id = "panel1_tab1",
                  wellPanel(title = "",
                            fluidRow(
                              column(4, actionButton("button_tab1","Check In Value",width='100%')
                              )
                            )
                  )
              ),

              div(id = "panel2_tab2",
                  wellPanel(title = "",

                            fluidRow(
                              column(8,
                                     textInput("valInput1", label = "Enter Input", value = "",width='100%')
                              )
                            )
                  )
              )
      ),

      # Second tab content
      tabItem(tabName = "tab2",

              div(id = "panel1_tab2",
                  wellPanel(title = "",
                            fluidRow(
                              column(4, actionButton("button_tab2","Check In Value",width='100%')
                              )
                            )
                  )
              ),

              div(id = "panel2_tab2",
                  wellPanel(title = "",
                            fluidRow(
                              column(8,
                                     textInput("valInput2", label = "Enter Input", value = "",width='100%')
                              )
                            )
                  )
              )
      )
    )
  )
)

My intent is that: in each tab, if the user clicks the button, the textInput box in the other tab should be removed completely. Here is my code in server.r

server <- function(input, output, session) {

  observeEvent(input$button_tab1,
               {
                 removeUI(selector = '#valInput2')
               }
  )

  observeEvent(input$button_tab2,
               {
                 removeUI(selector = '#valInput1')
               }
  )
}

However, although the textInput box gets removed, its label "Enter Input" still remains. How do remove this entire UI completely?

Thanks,