Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2380 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.4555. Min Range: -0.4555

Max Act: 0.4115. Min Act: -0.0002

Data Index: 616283 (C4 (Web Text))

Max Activating Token Index: 378

Text #1

Max Range: 0.4555. Min Range: -0.4555

Max Act: 0.4132. Min Act: -0.0002

Data Index: 296141 (C4 (Web Text))

Max Activating Token Index: 675

Text #2

Max Range: 0.4555. Min Range: -0.4555

Max Act: 0.3544. Min Act: -0.0002

Data Index: 107227 (C4 (Web Text))

Max Activating Token Index: 553

Text #3

Max Range: 0.4555. Min Range: -0.4555

Max Act: 0.3873. Min Act: -0.0002

Data Index: 1044545 (C4 (Web Text))

Max Activating Token Index: 317

Text #4

Max Range: 0.4555. Min Range: -0.4555

Max Act: 0.4286. Min Act: -0.0002

Data Index: 1278917 (C4 (Web Text))

Max Activating Token Index: 337

Text #5

Max Range: 0.4555. Min Range: -0.4555

Max Act: 0.3797. Min Act: -0.0002

Data Index: 1045569 (C4 (Web Text))

Max Activating Token Index: 369

Text #6

Max Range: 0.4555. Min Range: -0.4555

Max Act: 0.4555. Min Act: -0.0002

Data Index: 1318353 (C4 (Web Text))

Max Activating Token Index: 234

Text #7

Max Range: 0.4555. Min Range: -0.4555

Max Act: 0.3545. Min Act: -0.0002

Data Index: 125744 (C4 (Web Text))

Max Activating Token Index: 779

Text #8

Max Range: 0.4555. Min Range: -0.4555

Max Act: 0.3517. Min Act: -0.0002

Data Index: 752351 (C4 (Web Text))

Max Activating Token Index: 806

Text #9

Max Range: 0.4555. Min Range: -0.4555

Max Act: 0.3791. Min Act: -0.0002

Data Index: 997708 (C4 (Web Text))

Max Activating Token Index: 643

Text #10

Max Range: 0.4555. Min Range: -0.4555

Max Act: 0.3846. Min Act: -0.0002

Data Index: 1247256 (C4 (Web Text))

Max Activating Token Index: 754

Text #11

Max Range: 0.4555. Min Range: -0.4555

Max Act: 0.3400. Min Act: -0.0002

Data Index: 1242769 (C4 (Web Text))

Max Activating Token Index: 660

Text #12

Max Range: 0.4555. Min Range: -0.4555

Max Act: 0.3665. Min Act: -0.0002

Data Index: 1331361 (C4 (Web Text))

Max Activating Token Index: 312

Text #13

Max Range: 0.4555. Min Range: -0.4555

Max Act: 0.3421. Min Act: -0.0002

Data Index: 1089248 (C4 (Web Text))

Max Activating Token Index: 745

Text #14

Max Range: 0.4555. Min Range: -0.4555

Max Act: 0.3169. Min Act: -0.0002

Data Index: 1047131 (C4 (Web Text))

Max Activating Token Index: 424

Text #15

Max Range: 0.4555. Min Range: -0.4555

Max Act: 0.3302. Min Act: -0.0002

Data Index: 858357 (C4 (Web Text))

Max Activating Token Index: 237

Text #16

Max Range: 0.4555. Min Range: -0.4555

Max Act: 0.4147. Min Act: -0.0002

Data Index: 495158 (C4 (Web Text))

Max Activating Token Index: 760

Text #17

Max Range: 0.4555. Min Range: -0.4555

Max Act: 0.3659. Min Act: -0.0002

Data Index: 580988 (C4 (Web Text))

Max Activating Token Index: 117

Text #18

Max Range: 0.4555. Min Range: -0.4555

Max Act: 0.3454. Min Act: -0.0002

Data Index: 210986 (C4 (Web Text))

Max Activating Token Index: 206

Text #19

Max Range: 0.4555. Min Range: -0.4555

Max Act: 0.3954. Min Act: -0.0002

Data Index: 1200214 (C4 (Web Text))

Max Activating Token Index: 881