Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 383 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.4923. Min Range: -0.4923

Max Act: 0.4923. Min Act: -0.0000

Data Index: 313578 (C4 (Web Text))

Max Activating Token Index: 908

Text #1

Max Range: 0.4923. Min Range: -0.4923

Max Act: 0.4343. Min Act: -0.0000

Data Index: 379070 (C4 (Web Text))

Max Activating Token Index: 947

Text #2

Max Range: 0.4923. Min Range: -0.4923

Max Act: 0.3789. Min Act: -0.0000

Data Index: 416605 (C4 (Web Text))

Max Activating Token Index: 590

Text #3

Max Range: 0.4923. Min Range: -0.4923

Max Act: 0.3039. Min Act: -0.0000

Data Index: 902674 (C4 (Web Text))

Max Activating Token Index: 417

Text #4

Max Range: 0.4923. Min Range: -0.4923

Max Act: 0.2489. Min Act: -0.0000

Data Index: 918945 (C4 (Web Text))

Max Activating Token Index: 221

Text #5

Max Range: 0.4923. Min Range: -0.4923

Max Act: 0.2348. Min Act: -0.0000

Data Index: 883259 (C4 (Web Text))

Max Activating Token Index: 60

Text #6

Max Range: 0.4923. Min Range: -0.4923

Max Act: 0.2359. Min Act: -0.0000

Data Index: 1148571 (C4 (Web Text))

Max Activating Token Index: 488

Text #7

Max Range: 0.4923. Min Range: -0.4923

Max Act: 0.2254. Min Act: -0.0000

Data Index: 64726 (C4 (Web Text))

Max Activating Token Index: 270

Text #8

Max Range: 0.4923. Min Range: -0.4923

Max Act: 0.2326. Min Act: -0.0000

Data Index: 1159428 (C4 (Web Text))

Max Activating Token Index: 221

Text #9

Max Range: 0.4923. Min Range: -0.4923

Max Act: 0.1982. Min Act: -0.0000

Data Index: 1614030 (Python Code)

Max Activating Token Index: 692

Text #10

Max Range: 0.4923. Min Range: -0.4923

Max Act: 0.2609. Min Act: -0.0000

Data Index: 501589 (C4 (Web Text))

Max Activating Token Index: 615

Text #11

Max Range: 0.4923. Min Range: -0.4923

Max Act: 0.2418. Min Act: -0.0000

Data Index: 1128421 (C4 (Web Text))

Max Activating Token Index: 757

Text #12

Max Range: 0.4923. Min Range: -0.4923

Max Act: 0.2503. Min Act: -0.0000

Data Index: 749980 (C4 (Web Text))

Max Activating Token Index: 785

Text #13

Max Range: 0.4923. Min Range: -0.4923

Max Act: 0.2284. Min Act: -0.0000

Data Index: 1576756 (Python Code)

Max Activating Token Index: 405

Text #14

Max Range: 0.4923. Min Range: -0.4923

Max Act: 0.2153. Min Act: -0.0000

Data Index: 1390758 (Python Code)

Max Activating Token Index: 697

Text #15

Max Range: 0.4923. Min Range: -0.4923

Max Act: 0.2209. Min Act: -0.0000

Data Index: 223307 (C4 (Web Text))

Max Activating Token Index: 522

Text #16

Max Range: 0.4923. Min Range: -0.4923

Max Act: 0.2096. Min Act: -0.0000

Data Index: 188926 (C4 (Web Text))

Max Activating Token Index: 361

Text #17

Max Range: 0.4923. Min Range: -0.4923

Max Act: 0.2029. Min Act: -0.0000

Data Index: 151287 (C4 (Web Text))

Max Activating Token Index: 309

Text #18

Max Range: 0.4923. Min Range: -0.4923

Max Act: 0.2031. Min Act: -0.0000

Data Index: 595378 (C4 (Web Text))

Max Activating Token Index: 452

Text #19

Max Range: 0.4923. Min Range: -0.4923

Max Act: 0.2018. Min Act: -0.0000

Data Index: 172822 (C4 (Web Text))

Max Activating Token Index: 60