Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 970 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 4.5892. Min Range: -4.5892

Max Act: 4.1067. Min Act: -0.0000

Data Index: 386680 (C4 (Web Text))

Max Activating Token Index: 798

Text #1

Max Range: 4.5892. Min Range: -4.5892

Max Act: 4.5892. Min Act: -0.0000

Data Index: 311916 (C4 (Web Text))

Max Activating Token Index: 979

Text #2

Max Range: 4.5892. Min Range: -4.5892

Max Act: 4.3360. Min Act: -0.0000

Data Index: 1191454 (C4 (Web Text))

Max Activating Token Index: 356

Text #3

Max Range: 4.5892. Min Range: -4.5892

Max Act: 3.6178. Min Act: -0.0000

Data Index: 9107 (C4 (Web Text))

Max Activating Token Index: 124

Text #4

Max Range: 4.5892. Min Range: -4.5892

Max Act: 3.3742. Min Act: -0.0000

Data Index: 1356394 (C4 (Web Text))

Max Activating Token Index: 475

Text #5

Max Range: 4.5892. Min Range: -4.5892

Max Act: 3.1639. Min Act: -0.0000

Data Index: 582653 (C4 (Web Text))

Max Activating Token Index: 503

Text #6

Max Range: 4.5892. Min Range: -4.5892

Max Act: 3.4214. Min Act: -0.0000

Data Index: 72020 (C4 (Web Text))

Max Activating Token Index: 69

Text #7

Max Range: 4.5892. Min Range: -4.5892

Max Act: 3.3203. Min Act: -0.0000

Data Index: 1266225 (C4 (Web Text))

Max Activating Token Index: 786

Text #8

Max Range: 4.5892. Min Range: -4.5892

Max Act: 3.0010. Min Act: -0.0000

Data Index: 1115537 (C4 (Web Text))

Max Activating Token Index: 61

Text #9

Max Range: 4.5892. Min Range: -4.5892

Max Act: 2.8433. Min Act: -0.0000

Data Index: 784212 (C4 (Web Text))

Max Activating Token Index: 793

Text #10

Max Range: 4.5892. Min Range: -4.5892

Max Act: 2.8053. Min Act: -0.0000

Data Index: 1243197 (C4 (Web Text))

Max Activating Token Index: 115

Text #11

Max Range: 4.5892. Min Range: -4.5892

Max Act: 3.0702. Min Act: -0.0000

Data Index: 723356 (C4 (Web Text))

Max Activating Token Index: 521

Text #12

Max Range: 4.5892. Min Range: -4.5892

Max Act: 2.8180. Min Act: -0.0000

Data Index: 936761 (C4 (Web Text))

Max Activating Token Index: 649

Text #13

Max Range: 4.5892. Min Range: -4.5892

Max Act: 2.7850. Min Act: -0.0000

Data Index: 540751 (C4 (Web Text))

Max Activating Token Index: 87

Text #14

Max Range: 4.5892. Min Range: -4.5892

Max Act: 2.7680. Min Act: -0.0000

Data Index: 342795 (C4 (Web Text))

Max Activating Token Index: 45

Text #15

Max Range: 4.5892. Min Range: -4.5892

Max Act: 2.7281. Min Act: -0.0000

Data Index: 805178 (C4 (Web Text))

Max Activating Token Index: 597

Text #16

Max Range: 4.5892. Min Range: -4.5892

Max Act: 2.7274. Min Act: -0.0000

Data Index: 906417 (C4 (Web Text))

Max Activating Token Index: 110

Text #17

Max Range: 4.5892. Min Range: -4.5892

Max Act: 2.6826. Min Act: -0.0000

Data Index: 785949 (C4 (Web Text))

Max Activating Token Index: 997

Text #18

Max Range: 4.5892. Min Range: -4.5892

Max Act: 2.7000. Min Act: -0.0000

Data Index: 528869 (C4 (Web Text))

Max Activating Token Index: 1013

Text #19

Max Range: 4.5892. Min Range: -4.5892

Max Act: 2.6611. Min Act: -0.0000

Data Index: 343342 (C4 (Web Text))

Max Activating Token Index: 497