Wharton finance professor Jeremy Siegel is expecting some months of negative job growth later this year, and if the Federal Reserve doesn’t respond with rate cuts, the market could struggle, he said Wednesday. The central bank raised interest rates this month, but also hinted at a potential pause in hikes. For Siegel, that may not be enough if the economy turns. “The worry I have is the Fed is going to say … ‘We’re going to stay tight,'” Siegel said on CNBC’s ” Halftime Report .” “If we see payrolls go negative, if we see GDP negative. If the Fed doesn’t cut, then it’s going to be tougher sledding for the markets.” .SPX YTD mountain S & P 500 year to date Thanks to the banking crisis, restrictions on lending will come through the next few months, Seigel said. He expects that could slow down economic activity and cause negative payroll growth. Inflation has been showing signs of easing. The consumer price index for April , released Wednesday, rose 0.4% for the month, in line with expectations. However, the annual increase of 4.9%, was less than expected — and it was the slowest pace of growth since April 2021. “We know the tremendous lag in the housing sector will play itself out in the second half of this year,” Siegel said. Increases in housing costs was one of the factors that had pushed the index higher in April, along with gasoline and used vehicles. Siegel said he sees a “meaningful” gain in the stock market this year if Fed officials “will respond to the downside as rigorously as they responded to the upside.” If that happens, the S & P 500 could gain 15% total return, he said. However, he warned, if the central bank doesn’t respond quickly, he sees a more “muted” return of 5% to 10% for the year.