Remove stop words from searchstring in PHP

Question

I am facing problems with a php functin for optimizing a search string for a mssql query.

I need to find an entry which look like 'hobbit, the' by searching for 'the hobbit'. I thought about cutting the articles (in germany we have 'der', 'die' and 'das') if they have a trailing space out of the search string.

my function looks like:

      public function optimizeSearchString($searchString)
      {
        $articles = [
          'der ',
          'die ',
          'das ',
          'the '
        ];


        foreach ($articles as $article) {
//only cut $article out of $searchString if its longer than the $article itself
          if (strlen($searchString) > strlen($article) && strpos($searchString, $article)) {
            $searchString = str_replace($article, '', $searchString);
            break;
          }
        }

        return $searchString;
      }

but this doesn't work...

Maybe there is a nicer solution using regular expressions?

Answer 1

1.) To just remove one stopword from start or end of the string by using regex like this:

~^\W*(der|die|das|the)\W+\b|\b\W+(?1)\W*$~i

• ~ is the pattern delimiter
• ^ the caret anchor matches start of the string
• \W (upper) is a short for a character, that is not a word character
• (der|die|das|the) alternation | in first parenthesized group
• \b matches a word boundary
• At (?1) the pattern of first group is pasted
• $ matches right after the last character in the string
• Used i (PCRE_CASELESS) flag. If input is utf-8, also need u (PCRE_UTF8) flag.

Reference - What does this regex mean

Generate the pattern:

// array containing stopwords
$stopwords = array("der", "die", "das", "the");

// escape the stopword array and implode with pipe
$s = '~^\W*('.implode("|", array_map("preg_quote", $stopwords)).')\W+\b|\b\W+(?1)\W*$~i';

// replace with emptystring
$searchString = preg_replace($s, "", $searchString);

Note that if ~ delimiter occurs in the $stopwords array, it also has to be escaped with a backslash.

PHP test at eval.in, Regex pattern at regex101

---

2.) But to remove stop words anywhere in the string how about splitting into words:

// words to be removed
$stopwords = array(
'der' => 1,
'die' => 1,
'das' => 1,
'the' => 1);
# used words as key for better performance

// remove stopwords from string
function strip_stopwords($str = "")
{
  global $stopwords;

  // 1.) break string into words
  // [^-\w\'] matches characters, that are not [0-9a-zA-Z_-']
  // if input is unicode/utf-8, the u flag is needed: /pattern/u
  $words = preg_split('/[^-\w\']+/', $str, -1, PREG_SPLIT_NO_EMPTY);

  // 2.) if we have at least 2 words, remove stopwords
  if(count($words) > 1)
  {
    $words = array_filter($words, function ($w) use (&$stopwords) {
      return !isset($stopwords[strtolower($w)]);
      # if utf-8: mb_strtolower($w, "utf-8")
    });
  }

  // check if not too much was removed such as "the the" would return empty
  if(!empty($words))
    return implode(" ", $words);
  return $str;
}

See demo at eval.in, ideone.com

// test it
echo strip_stopwords("The Hobbit das foo, der");

Hobbit foo

This solution will also remove any punctuation besides _ - ' because it's imploding remaining words with space after removing the common words. The idea is to prepare the string for a query.

Both solutions don't modify the case and will leave the string if it only consists of one stopword.

Lists of common words

• Most common words in English ^Wikipedia
• Most frequent words in German language ^Wikipedia
• MySQL: English full-text stopwords
• Default English stopwords list
• List of German stopwords

Answer 2

The solution provided by @Jonny 5 seems to be the best for my solution.

Now I use a function like this:

  public function optimizeSearchString($searchString = "")
  {
    $stopwords = array(
      'der' => 1,
      'die' => 1,
      'das' => 1,
      'the' => 1);

    $words = preg_split('/[^-\w\']+/', $searchString, -1, PREG_SPLIT_NO_EMPTY);

    if (count($words) > 1) {
      $words = array_filter($words, function ($v) use (&$stopwords) {
        return !isset($stopwords[strtolower($v)]);
      }
      );
    }

    if (empty($words)) {
      return $searchString;
    }

    return implode(" ", $words);
  }

The new solution of Jonny 5 would work also, but I use this one, because I'm not that familiar with regex and I know whats going on :-)

Answer 3

This is what i do.

public function optimizeSearchString($searchString) {
    $wordsFromSearchString = str_word_count($searchString, true);
    $finalWords = array_diff($wordsFromSearchString, $stopwords);
    return implode(" ", $finalWords);
}

Answer 4

I made a different version using array_diff, which @Yashrajsinh Jadeja also did. I added a third parameter 'strcasecmp' to ignore case and made the input an array using a simple word tokenizer.

//Search string with article
$searchString = "Das blaue Haus"; //"The blue house"

//Split string into array. (This method is insufficient and doesn't account for compound nouns like "blue jay" or "einfamilienhaus".)
$wordArray = preg_split('/[^-\w\']+/', $searchString, -1, PREG_SPLIT_NO_EMPTY); 

var_dump(optimizeSearchString($wordArray));

function optimizeSearchString($wordArray) {
  $articles = array('der', 'die', 'das', 'the');
  $newArray = array_udiff($wordArray, $articles, 'strcasecmp');
  return $newArray;
}

Output:

array(2) {
  [1]=>
  string(5) "blaue"
  [2]=>
  string(4) "Haus"
}

Answer 5

public function optimizeSearchString($searchString)
{
        $articles = (
          'der ',
          'die ',
          'das ',
          'the '
        );


        foreach ($articles as $article) {
         //only cut $article out of $searchString if its longer than the $article itself
          if (strlen($searchString) > strlen($article) && strpos($searchString, $article)) {
            $searchString = str_replace($article, '', $searchString);
            break;
          }
        }

        return $searchString;
}