Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2118 in Layer 4

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 5.3875. Min Range: -5.3875

Max Act: 5.3875. Min Act: -0.0001

Data Index: 1262935 (C4 (Web Text))

Max Activating Token Index: 498

Text #1

Max Range: 5.3875. Min Range: -5.3875

Max Act: 4.6979. Min Act: -0.0000

Data Index: 326692 (C4 (Web Text))

Max Activating Token Index: 635

Text #2

Max Range: 5.3875. Min Range: -5.3875

Max Act: 4.8597. Min Act: -0.0001

Data Index: 952553 (C4 (Web Text))

Max Activating Token Index: 135

Text #3

Max Range: 5.3875. Min Range: -5.3875

Max Act: 4.8016. Min Act: -0.0001

Data Index: 388058 (C4 (Web Text))

Max Activating Token Index: 415

Text #4

Max Range: 5.3875. Min Range: -5.3875

Max Act: 4.6663. Min Act: -0.0001

Data Index: 1089125 (C4 (Web Text))

Max Activating Token Index: 364

Text #5

Max Range: 5.3875. Min Range: -5.3875

Max Act: 4.6197. Min Act: -0.0000

Data Index: 1356218 (C4 (Web Text))

Max Activating Token Index: 259

Text #6

Max Range: 5.3875. Min Range: -5.3875

Max Act: 4.5463. Min Act: -0.0000

Data Index: 1169369 (C4 (Web Text))

Max Activating Token Index: 546

Text #7

Max Range: 5.3875. Min Range: -5.3875

Max Act: 4.5651. Min Act: -0.0001

Data Index: 993675 (C4 (Web Text))

Max Activating Token Index: 168

Text #8

Max Range: 5.3875. Min Range: -5.3875

Max Act: 4.6923. Min Act: -0.0001

Data Index: 697129 (C4 (Web Text))

Max Activating Token Index: 260

Text #9

Max Range: 5.3875. Min Range: -5.3875

Max Act: 4.2871. Min Act: -0.0001

Data Index: 610651 (C4 (Web Text))

Max Activating Token Index: 503

Text #10

Max Range: 5.3875. Min Range: -5.3875

Max Act: 4.7842. Min Act: -0.0001

Data Index: 981021 (C4 (Web Text))

Max Activating Token Index: 834

Text #11

Max Range: 5.3875. Min Range: -5.3875

Max Act: 4.4151. Min Act: -0.0001

Data Index: 1273318 (C4 (Web Text))

Max Activating Token Index: 397

Text #12

Max Range: 5.3875. Min Range: -5.3875

Max Act: 4.1796. Min Act: -0.0001

Data Index: 26812 (C4 (Web Text))

Max Activating Token Index: 45

Text #13

Max Range: 5.3875. Min Range: -5.3875

Max Act: 4.3503. Min Act: -0.0001

Data Index: 576348 (C4 (Web Text))

Max Activating Token Index: 200

Text #14

Max Range: 5.3875. Min Range: -5.3875

Max Act: 4.4110. Min Act: -0.0001

Data Index: 656213 (C4 (Web Text))

Max Activating Token Index: 896

Text #15

Max Range: 5.3875. Min Range: -5.3875

Max Act: 4.0887. Min Act: -0.0001

Data Index: 563027 (C4 (Web Text))

Max Activating Token Index: 952

Text #16

Max Range: 5.3875. Min Range: -5.3875

Max Act: 3.9697. Min Act: -0.0000

Data Index: 532041 (C4 (Web Text))

Max Activating Token Index: 788

Text #17

Max Range: 5.3875. Min Range: -5.3875

Max Act: 3.6530. Min Act: -0.0001

Data Index: 651834 (C4 (Web Text))

Max Activating Token Index: 497

Text #18

Max Range: 5.3875. Min Range: -5.3875

Max Act: 4.0723. Min Act: -0.0001

Data Index: 1238484 (C4 (Web Text))

Max Activating Token Index: 562

Text #19

Max Range: 5.3875. Min Range: -5.3875

Max Act: 3.7564. Min Act: -0.0001

Data Index: 965226 (C4 (Web Text))

Max Activating Token Index: 168