Model: SoLU Model: 8 Layers, 4096 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 49 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-8l')

Text #0

Max Range: 0.8226. Min Range: -0.8226

Max Act: 0.8226. Min Act: -0.0001

Data Index: 136117 (C4 (Web Text))

Max Activating Token Index: 141

Text #1

Max Range: 0.8226. Min Range: -0.8226

Max Act: 0.7645. Min Act: -0.0001

Data Index: 969790 (C4 (Web Text))

Max Activating Token Index: 708

Text #2

Max Range: 0.8226. Min Range: -0.8226

Max Act: 0.7731. Min Act: -0.0001

Data Index: 347647 (C4 (Web Text))

Max Activating Token Index: 540

Text #3

Max Range: 0.8226. Min Range: -0.8226

Max Act: 0.6932. Min Act: -0.0001

Data Index: 593263 (C4 (Web Text))

Max Activating Token Index: 779

Text #4

Max Range: 0.8226. Min Range: -0.8226

Max Act: 0.7710. Min Act: -0.0001

Data Index: 566530 (C4 (Web Text))

Max Activating Token Index: 595

Text #5

Max Range: 0.8226. Min Range: -0.8226

Max Act: 0.6498. Min Act: -0.0001

Data Index: 1114418 (C4 (Web Text))

Max Activating Token Index: 495

Text #6

Max Range: 0.8226. Min Range: -0.8226

Max Act: 0.6636. Min Act: -0.0001

Data Index: 889558 (C4 (Web Text))

Max Activating Token Index: 457

Text #7

Max Range: 0.8226. Min Range: -0.8226

Max Act: 0.6508. Min Act: -0.0001

Data Index: 705078 (C4 (Web Text))

Max Activating Token Index: 153

Text #8

Max Range: 0.8226. Min Range: -0.8226

Max Act: 0.6722. Min Act: -0.0001

Data Index: 408484 (C4 (Web Text))

Max Activating Token Index: 814

Text #9

Max Range: 0.8226. Min Range: -0.8226

Max Act: 0.6458. Min Act: -0.0001

Data Index: 410417 (C4 (Web Text))

Max Activating Token Index: 352

Text #10

Max Range: 0.8226. Min Range: -0.8226

Max Act: 0.6028. Min Act: -0.0001

Data Index: 1211189 (C4 (Web Text))

Max Activating Token Index: 925

Text #11

Max Range: 0.8226. Min Range: -0.8226

Max Act: 0.6026. Min Act: -0.0001

Data Index: 608662 (C4 (Web Text))

Max Activating Token Index: 900

Text #12

Max Range: 0.8226. Min Range: -0.8226

Max Act: 0.6078. Min Act: -0.0001

Data Index: 968496 (C4 (Web Text))

Max Activating Token Index: 591

Text #13

Max Range: 0.8226. Min Range: -0.8226

Max Act: 0.5960. Min Act: -0.0001

Data Index: 766016 (C4 (Web Text))

Max Activating Token Index: 827

Text #14

Max Range: 0.8226. Min Range: -0.8226

Max Act: 0.5626. Min Act: -0.0001

Data Index: 578399 (C4 (Web Text))

Max Activating Token Index: 757

Text #15

Max Range: 0.8226. Min Range: -0.8226

Max Act: 0.5799. Min Act: -0.0001

Data Index: 213514 (C4 (Web Text))

Max Activating Token Index: 409

Text #16

Max Range: 0.8226. Min Range: -0.8226

Max Act: 0.5764. Min Act: -0.0001

Data Index: 767321 (C4 (Web Text))

Max Activating Token Index: 593

Text #17

Max Range: 0.8226. Min Range: -0.8226

Max Act: 0.5113. Min Act: -0.0001

Data Index: 944430 (C4 (Web Text))

Max Activating Token Index: 758

Text #18

Max Range: 0.8226. Min Range: -0.8226

Max Act: 0.5090. Min Act: -0.0001

Data Index: 1074464 (C4 (Web Text))

Max Activating Token Index: 124

Text #19

Max Range: 0.8226. Min Range: -0.8226

Max Act: 0.5525. Min Act: -0.0001

Data Index: 296263 (C4 (Web Text))

Max Activating Token Index: 443