Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 113 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.6056. Min Range: -0.6056

Max Act: 0.6056. Min Act: -0.0000

Data Index: 456793 (C4 (Web Text))

Max Activating Token Index: 210

Text #1

Max Range: 0.6056. Min Range: -0.6056

Max Act: 0.5807. Min Act: -0.0000

Data Index: 1390657 (Python Code)

Max Activating Token Index: 463

Text #2

Max Range: 0.6056. Min Range: -0.6056

Max Act: 0.5527. Min Act: -0.0000

Data Index: 873288 (C4 (Web Text))

Max Activating Token Index: 263

Text #3

Max Range: 0.6056. Min Range: -0.6056

Max Act: 0.4985. Min Act: -0.0000

Data Index: 869803 (C4 (Web Text))

Max Activating Token Index: 796

Text #4

Max Range: 0.6056. Min Range: -0.6056

Max Act: 0.5318. Min Act: -0.0000

Data Index: 897119 (C4 (Web Text))

Max Activating Token Index: 329

Text #5

Max Range: 0.6056. Min Range: -0.6056

Max Act: 0.5205. Min Act: -0.0000

Data Index: 1526337 (Python Code)

Max Activating Token Index: 524

Text #6

Max Range: 0.6056. Min Range: -0.6056

Max Act: 0.4883. Min Act: -0.0000

Data Index: 230880 (C4 (Web Text))

Max Activating Token Index: 50

Text #7

Max Range: 0.6056. Min Range: -0.6056

Max Act: 0.4710. Min Act: -0.0000

Data Index: 1443703 (Python Code)

Max Activating Token Index: 575

Text #8

Max Range: 0.6056. Min Range: -0.6056

Max Act: 0.4811. Min Act: -0.0000

Data Index: 1567018 (Python Code)

Max Activating Token Index: 436

Text #9

Max Range: 0.6056. Min Range: -0.6056

Max Act: 0.4638. Min Act: -0.0000

Data Index: 598451 (C4 (Web Text))

Max Activating Token Index: 708

Text #10

Max Range: 0.6056. Min Range: -0.6056

Max Act: 0.4615. Min Act: -0.0000

Data Index: 782646 (C4 (Web Text))

Max Activating Token Index: 406

Text #11

Max Range: 0.6056. Min Range: -0.6056

Max Act: 0.4442. Min Act: -0.0000

Data Index: 1454316 (Python Code)

Max Activating Token Index: 765

Text #12

Max Range: 0.6056. Min Range: -0.6056

Max Act: 0.4081. Min Act: -0.0000

Data Index: 1581892 (Python Code)

Max Activating Token Index: 942

Text #13

Max Range: 0.6056. Min Range: -0.6056

Max Act: 0.4040. Min Act: -0.0000

Data Index: 302613 (C4 (Web Text))

Max Activating Token Index: 278

Text #14

Max Range: 0.6056. Min Range: -0.6056

Max Act: 0.3811. Min Act: -0.0000

Data Index: 591447 (C4 (Web Text))

Max Activating Token Index: 125

Text #15

Max Range: 0.6056. Min Range: -0.6056

Max Act: 0.4128. Min Act: -0.0000

Data Index: 1189496 (C4 (Web Text))

Max Activating Token Index: 590

Text #16

Max Range: 0.6056. Min Range: -0.6056

Max Act: 0.4150. Min Act: -0.0000

Data Index: 330995 (C4 (Web Text))

Max Activating Token Index: 177

Text #17

Max Range: 0.6056. Min Range: -0.6056

Max Act: 0.4094. Min Act: -0.0000

Data Index: 825022 (C4 (Web Text))

Max Activating Token Index: 559

Text #18

Max Range: 0.6056. Min Range: -0.6056

Max Act: 0.4035. Min Act: -0.0000

Data Index: 1418456 (Python Code)

Max Activating Token Index: 433

Text #19

Max Range: 0.6056. Min Range: -0.6056

Max Act: 0.3989. Min Act: -0.0000

Data Index: 368574 (C4 (Web Text))

Max Activating Token Index: 471