Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3668 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.6109. Min Range: -0.6109

Max Act: 0.4963. Min Act: -0.0002

Data Index: 1180768 (C4 (Web Text))

Max Activating Token Index: 685

Text #1

Max Range: 0.6109. Min Range: -0.6109

Max Act: 0.4949. Min Act: -0.0002

Data Index: 969281 (C4 (Web Text))

Max Activating Token Index: 873

Text #2

Max Range: 0.6109. Min Range: -0.6109

Max Act: 0.6109. Min Act: -0.0002

Data Index: 1331112 (C4 (Web Text))

Max Activating Token Index: 918

Text #3

Max Range: 0.6109. Min Range: -0.6109

Max Act: 0.4913. Min Act: -0.0002

Data Index: 1245827 (C4 (Web Text))

Max Activating Token Index: 345

Text #4

Max Range: 0.6109. Min Range: -0.6109

Max Act: 0.4480. Min Act: -0.0002

Data Index: 949355 (C4 (Web Text))

Max Activating Token Index: 783

Text #5

Max Range: 0.6109. Min Range: -0.6109

Max Act: 0.4769. Min Act: -0.0002

Data Index: 916586 (C4 (Web Text))

Max Activating Token Index: 514

Text #6

Max Range: 0.6109. Min Range: -0.6109

Max Act: 0.5107. Min Act: -0.0002

Data Index: 286697 (C4 (Web Text))

Max Activating Token Index: 961

Text #7

Max Range: 0.6109. Min Range: -0.6109

Max Act: 0.5108. Min Act: -0.0002

Data Index: 129074 (C4 (Web Text))

Max Activating Token Index: 174

Text #8

Max Range: 0.6109. Min Range: -0.6109

Max Act: 0.4482. Min Act: -0.0002

Data Index: 390665 (C4 (Web Text))

Max Activating Token Index: 753

Text #9

Max Range: 0.6109. Min Range: -0.6109

Max Act: 0.4163. Min Act: -0.0002

Data Index: 1208337 (C4 (Web Text))

Max Activating Token Index: 823

Text #10

Max Range: 0.6109. Min Range: -0.6109

Max Act: 0.4701. Min Act: -0.0002

Data Index: 215079 (C4 (Web Text))

Max Activating Token Index: 634

Text #11

Max Range: 0.6109. Min Range: -0.6109

Max Act: 0.4224. Min Act: -0.0002

Data Index: 650908 (C4 (Web Text))

Max Activating Token Index: 367

Text #12

Max Range: 0.6109. Min Range: -0.6109

Max Act: 0.4607. Min Act: -0.0002

Data Index: 312243 (C4 (Web Text))

Max Activating Token Index: 734

Text #13

Max Range: 0.6109. Min Range: -0.6109

Max Act: 0.4431. Min Act: -0.0002

Data Index: 478262 (C4 (Web Text))

Max Activating Token Index: 975

Text #14

Max Range: 0.6109. Min Range: -0.6109

Max Act: 0.4565. Min Act: -0.0002

Data Index: 942659 (C4 (Web Text))

Max Activating Token Index: 673

Text #15

Max Range: 0.6109. Min Range: -0.6109

Max Act: 0.4230. Min Act: -0.0002

Data Index: 1298931 (C4 (Web Text))

Max Activating Token Index: 306

Text #16

Max Range: 0.6109. Min Range: -0.6109

Max Act: 0.4494. Min Act: -0.0002

Data Index: 892896 (C4 (Web Text))

Max Activating Token Index: 429

Text #17

Max Range: 0.6109. Min Range: -0.6109

Max Act: 0.5357. Min Act: -0.0002

Data Index: 1103945 (C4 (Web Text))

Max Activating Token Index: 579

Text #18

Max Range: 0.6109. Min Range: -0.6109

Max Act: 0.5363. Min Act: -0.0002

Data Index: 802852 (C4 (Web Text))

Max Activating Token Index: 546

Text #19

Max Range: 0.6109. Min Range: -0.6109

Max Act: 0.4663. Min Act: -0.0002

Data Index: 543349 (C4 (Web Text))

Max Activating Token Index: 932