Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1106 in Layer 10

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.9793. Min Range: -2.9793

Max Act: 2.9793. Min Act: -0.0000

Data Index: 170981 (C4 (Web Text))

Max Activating Token Index: 638

Text #1

Max Range: 2.9793. Min Range: -2.9793

Max Act: 2.7661. Min Act: -0.0000

Data Index: 1359783 (C4 (Web Text))

Max Activating Token Index: 921

Text #2

Max Range: 2.9793. Min Range: -2.9793

Max Act: 2.7659. Min Act: -0.0000

Data Index: 22551 (C4 (Web Text))

Max Activating Token Index: 549

Text #3

Max Range: 2.9793. Min Range: -2.9793

Max Act: 2.5852. Min Act: -0.0000

Data Index: 489376 (C4 (Web Text))

Max Activating Token Index: 918

Text #4

Max Range: 2.9793. Min Range: -2.9793

Max Act: 2.7030. Min Act: -0.0000

Data Index: 970918 (C4 (Web Text))

Max Activating Token Index: 674

Text #5

Max Range: 2.9793. Min Range: -2.9793

Max Act: 2.5627. Min Act: -0.0000

Data Index: 274997 (C4 (Web Text))

Max Activating Token Index: 876

Text #6

Max Range: 2.9793. Min Range: -2.9793

Max Act: 2.4562. Min Act: -0.0000

Data Index: 547425 (C4 (Web Text))

Max Activating Token Index: 453

Text #7

Max Range: 2.9793. Min Range: -2.9793

Max Act: 2.5406. Min Act: -0.0000

Data Index: 743935 (C4 (Web Text))

Max Activating Token Index: 703

Text #8

Max Range: 2.9793. Min Range: -2.9793

Max Act: 2.5285. Min Act: -0.0000

Data Index: 496668 (C4 (Web Text))

Max Activating Token Index: 599

Text #9

Max Range: 2.9793. Min Range: -2.9793

Max Act: 2.3718. Min Act: -0.0000

Data Index: 710851 (C4 (Web Text))

Max Activating Token Index: 762

Text #10

Max Range: 2.9793. Min Range: -2.9793

Max Act: 2.2970. Min Act: -0.0000

Data Index: 266057 (C4 (Web Text))

Max Activating Token Index: 984

Text #11

Max Range: 2.9793. Min Range: -2.9793

Max Act: 2.2238. Min Act: -0.0000

Data Index: 615910 (C4 (Web Text))

Max Activating Token Index: 94

Text #12

Max Range: 2.9793. Min Range: -2.9793

Max Act: 2.3059. Min Act: -0.0000

Data Index: 1174824 (C4 (Web Text))

Max Activating Token Index: 1021

Text #13

Max Range: 2.9793. Min Range: -2.9793

Max Act: 2.3087. Min Act: -0.0000

Data Index: 1192332 (C4 (Web Text))

Max Activating Token Index: 872

Text #14

Max Range: 2.9793. Min Range: -2.9793

Max Act: 1.9869. Min Act: -0.0000

Data Index: 1262601 (C4 (Web Text))

Max Activating Token Index: 368

Text #15

Max Range: 2.9793. Min Range: -2.9793

Max Act: 2.0884. Min Act: -0.0000

Data Index: 194130 (C4 (Web Text))

Max Activating Token Index: 234

Text #16

Max Range: 2.9793. Min Range: -2.9793

Max Act: 2.0041. Min Act: -0.0000

Data Index: 934634 (C4 (Web Text))

Max Activating Token Index: 828

Text #17

Max Range: 2.9793. Min Range: -2.9793

Max Act: 1.9535. Min Act: -0.0000

Data Index: 449836 (C4 (Web Text))

Max Activating Token Index: 956

Text #18

Max Range: 2.9793. Min Range: -2.9793

Max Act: 2.0328. Min Act: -0.0000

Data Index: 560712 (C4 (Web Text))

Max Activating Token Index: 845

Text #19

Max Range: 2.9793. Min Range: -2.9793

Max Act: 2.0326. Min Act: -0.0000

Data Index: 19956 (C4 (Web Text))

Max Activating Token Index: 540