Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 893 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.6538. Min Range: -0.6538

Max Act: 0.6333. Min Act: -0.0000

Data Index: 924438 (C4 (Web Text))

Max Activating Token Index: 691

Text #1

Max Range: 0.6538. Min Range: -0.6538

Max Act: 0.6538. Min Act: -0.0000

Data Index: 1011897 (C4 (Web Text))

Max Activating Token Index: 199

Text #2

Max Range: 0.6538. Min Range: -0.6538

Max Act: 0.6226. Min Act: -0.0000

Data Index: 670642 (C4 (Web Text))

Max Activating Token Index: 783

Text #3

Max Range: 0.6538. Min Range: -0.6538

Max Act: 0.5410. Min Act: -0.0000

Data Index: 1273702 (C4 (Web Text))

Max Activating Token Index: 1022

Text #4

Max Range: 0.6538. Min Range: -0.6538

Max Act: 0.5123. Min Act: -0.0000

Data Index: 1268229 (C4 (Web Text))

Max Activating Token Index: 231

Text #5

Max Range: 0.6538. Min Range: -0.6538

Max Act: 0.5585. Min Act: -0.0000

Data Index: 257152 (C4 (Web Text))

Max Activating Token Index: 543

Text #6

Max Range: 0.6538. Min Range: -0.6538

Max Act: 0.5599. Min Act: -0.0000

Data Index: 1124412 (C4 (Web Text))

Max Activating Token Index: 902

Text #7

Max Range: 0.6538. Min Range: -0.6538

Max Act: 0.5371. Min Act: -0.0000

Data Index: 344553 (C4 (Web Text))

Max Activating Token Index: 732

Text #8

Max Range: 0.6538. Min Range: -0.6538

Max Act: 0.5766. Min Act: -0.0000

Data Index: 1060805 (C4 (Web Text))

Max Activating Token Index: 100

Text #9

Max Range: 0.6538. Min Range: -0.6538

Max Act: 0.4912. Min Act: -0.0000

Data Index: 1102034 (C4 (Web Text))

Max Activating Token Index: 720

Text #10

Max Range: 0.6538. Min Range: -0.6538

Max Act: 0.5083. Min Act: -0.0000

Data Index: 832606 (C4 (Web Text))

Max Activating Token Index: 722

Text #11

Max Range: 0.6538. Min Range: -0.6538

Max Act: 0.5075. Min Act: -0.0000

Data Index: 654437 (C4 (Web Text))

Max Activating Token Index: 317

Text #12

Max Range: 0.6538. Min Range: -0.6538

Max Act: 0.4919. Min Act: -0.0000

Data Index: 646236 (C4 (Web Text))

Max Activating Token Index: 865

Text #13

Max Range: 0.6538. Min Range: -0.6538

Max Act: 0.5265. Min Act: -0.0000

Data Index: 684906 (C4 (Web Text))

Max Activating Token Index: 131

Text #14

Max Range: 0.6538. Min Range: -0.6538

Max Act: 0.5164. Min Act: -0.0000

Data Index: 440880 (C4 (Web Text))

Max Activating Token Index: 900

Text #15

Max Range: 0.6538. Min Range: -0.6538

Max Act: 0.5252. Min Act: -0.0000

Data Index: 542317 (C4 (Web Text))

Max Activating Token Index: 168

Text #16

Max Range: 0.6538. Min Range: -0.6538

Max Act: 0.4548. Min Act: -0.0000

Data Index: 480983 (C4 (Web Text))

Max Activating Token Index: 748

Text #17

Max Range: 0.6538. Min Range: -0.6538

Max Act: 0.3778. Min Act: -0.0000

Data Index: 805401 (C4 (Web Text))

Max Activating Token Index: 151

Text #18

Max Range: 0.6538. Min Range: -0.6538

Max Act: 0.4035. Min Act: -0.0000

Data Index: 984802 (C4 (Web Text))

Max Activating Token Index: 376

Text #19

Max Range: 0.6538. Min Range: -0.6538

Max Act: 0.3832. Min Act: -0.0000

Data Index: 51006 (C4 (Web Text))

Max Activating Token Index: 763