Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5243 in Layer 11

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.9358. Min Range: -2.9358

Max Act: 2.9358. Min Act: -0.0000

Data Index: 1229609 (C4 (Web Text))

Max Activating Token Index: 428

Text #1

Max Range: 2.9358. Min Range: -2.9358

Max Act: 2.6967. Min Act: -0.0000

Data Index: 303273 (C4 (Web Text))

Max Activating Token Index: 292

Text #2

Max Range: 2.9358. Min Range: -2.9358

Max Act: 2.5371. Min Act: -0.0000

Data Index: 260830 (C4 (Web Text))

Max Activating Token Index: 284

Text #3

Max Range: 2.9358. Min Range: -2.9358

Max Act: 2.7851. Min Act: -0.0000

Data Index: 1170981 (C4 (Web Text))

Max Activating Token Index: 516

Text #4

Max Range: 2.9358. Min Range: -2.9358

Max Act: 2.5856. Min Act: -0.0000

Data Index: 1327647 (C4 (Web Text))

Max Activating Token Index: 258

Text #5

Max Range: 2.9358. Min Range: -2.9358

Max Act: 2.5979. Min Act: -0.0000

Data Index: 705099 (C4 (Web Text))

Max Activating Token Index: 440

Text #6

Max Range: 2.9358. Min Range: -2.9358

Max Act: 2.4269. Min Act: -0.0000

Data Index: 289616 (C4 (Web Text))

Max Activating Token Index: 212

Text #7

Max Range: 2.9358. Min Range: -2.9358

Max Act: 2.5486. Min Act: -0.0000

Data Index: 1227874 (C4 (Web Text))

Max Activating Token Index: 257

Text #8

Max Range: 2.9358. Min Range: -2.9358

Max Act: 2.2669. Min Act: -0.0000

Data Index: 851799 (C4 (Web Text))

Max Activating Token Index: 852

Text #9

Max Range: 2.9358. Min Range: -2.9358

Max Act: 2.6446. Min Act: -0.0000

Data Index: 258510 (C4 (Web Text))

Max Activating Token Index: 203

Text #10

Max Range: 2.9358. Min Range: -2.9358

Max Act: 2.6634. Min Act: -0.0000

Data Index: 20490 (C4 (Web Text))

Max Activating Token Index: 254

Text #11

Max Range: 2.9358. Min Range: -2.9358

Max Act: 2.4798. Min Act: -0.0000

Data Index: 532656 (C4 (Web Text))

Max Activating Token Index: 299

Text #12

Max Range: 2.9358. Min Range: -2.9358

Max Act: 2.2612. Min Act: -0.0000

Data Index: 1252018 (C4 (Web Text))

Max Activating Token Index: 79

Text #13

Max Range: 2.9358. Min Range: -2.9358

Max Act: 2.5562. Min Act: -0.0000

Data Index: 1095984 (C4 (Web Text))

Max Activating Token Index: 314

Text #14

Max Range: 2.9358. Min Range: -2.9358

Max Act: 2.4351. Min Act: -0.0000

Data Index: 1345033 (C4 (Web Text))

Max Activating Token Index: 186

Text #15

Max Range: 2.9358. Min Range: -2.9358

Max Act: 2.4252. Min Act: -0.0000

Data Index: 324983 (C4 (Web Text))

Max Activating Token Index: 261

Text #16

Max Range: 2.9358. Min Range: -2.9358

Max Act: 2.4168. Min Act: -0.0000

Data Index: 205209 (C4 (Web Text))

Max Activating Token Index: 243

Text #17

Max Range: 2.9358. Min Range: -2.9358

Max Act: 2.3864. Min Act: -0.0000

Data Index: 296143 (C4 (Web Text))

Max Activating Token Index: 269

Text #18

Max Range: 2.9358. Min Range: -2.9358

Max Act: 2.4122. Min Act: -0.0000

Data Index: 863627 (C4 (Web Text))

Max Activating Token Index: 111

Text #19

Max Range: 2.9358. Min Range: -2.9358

Max Act: 2.5147. Min Act: -0.0000

Data Index: 526831 (C4 (Web Text))

Max Activating Token Index: 983