The virulence factor concept has been a powerful engine in driving research and the intellectual flow in the fields of microbial pathogenesis and infectious diseases. This review analyzes virulence factors from the viewpoint of the damage–response framework of microbial pathogenesis, which defines virulence factor as microbial components that can damage a susceptible host. At a practical level, the finding that effective immune responses often target virulence factors provides a roadmap for future vaccine design. However, there are significant limitations to this concept, which are rooted in the inability to define virulence and virulence factors in the absence of host factors and the host response. In fact, this concept appears to work best for certain types of bacterial pathogens, being less well suited for viruses and commensal organisms with pathogenic potential.